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(54) Method and apparatus for adding ornaments to an image of a person 



(57) The position of a body part area for a personal 
image in an input image is detected. According to the 
position of the detected body part area, the position of 
the origin of a coordinate system for an ornament is de- 
fined. Based on the position of the defined origin, an or- 
nament-arranged input image is outputted. When the 
personal image moves in the input image, the ornament 
also moves following the movement of the personal im- 
age. Even when both personal image and ornament 
move, the ornament can be made not to interfere with 
the personal image. Therefore, the personal image can 
be clearly displayed. Moreover, the input image can be 
made look more interesting by synchronizing the move- 
ment of the ornament with the movement of the personal 
image. 
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Description 

[0001 J The present invention relates to an image processing apparatus built into a communication device (for exam- 
ple, a TV phone, a TV conference system, a video mail system, a video chat system, an intercom, etc.), handling image 
5 data, more particularly relates to an image processing apparatus editing a personal image (for example, a face image 
of a user, a whole-body image of the user, etc.). 

[0002] There is an amusement apparatus that prints and outputs a personal image of the user to various kinds of 
print media. The amusement apparatus laps a frame of ornaments (for example, a picture, a pattern, etc.) over the 
personal image to make a composite image, and prints and outputs the composite image to a print media (for example, 
10 seal etc.). 

[0003] The amusement apparatus does not have a function to adjust a positional relationship of the personal image 
and the ornaments, in itself. Therefore, even if the user has arranged the personal image to fit in the ornament frame, 
the ornament may overlap with the personal image, so that the ornament hides a face part of the user and/or a head 
part of the user and the personal image becomes unclear. 

15 [0004] Considering this point, a reference 1 (published Japanese Patent Application Laid-Open No. 2000-22929) 
discloses a technique that adds ornaments to a persona! image, while making an ornament not overlap with a body 
part of the personal image, especially, a predetermined body part corresponding to a face and heads. 
[0005] Recently, a camera-built-in cellular phone is put in practical use. The cellular phone has a function to lap a 
frame of ornaments over a personal image to make a composite image and a function to transmit the composite image. 

20 if a transmitter uses such a camera-built-in cellular phone, a receiver can enjoy seeing the picture that the transmitter 
has edited. 

[0006] Reference 1 assumes that ornaments do not move. According to the technique disclosed by reference 1 , 
ornaments are added to a personal image such that the ornaments do not overlap with a body part of the personal 
image, especially, a predetermined body part area corresponding to a face and heads. 

25 [0007] Consideration of reference 1 may be appropriate, since, in reference 1 , output is designated only to a printer 
that prints the composite image to a print media and an image once printed on the print media is rarely edited again. 
[0008] However, the camera-built-in cellular phones mentioned above have spread. In one of these cellular phones, 
a composite image is displayed on a display, without being printed to a print media. In the future, the user of one of 
the above-mentioned cellular phones will think that he/she wants to move ornaments in order to obtain more pleasure. 

30 [0009] However, in prior art, adding ornaments which do not move to a personal image which does not move as well 
is only taken into consideration. Adding ornaments which move to the personal image is not taken into consideration 
at all. For this reason, if prior art is simply applied to a personal image which moves, the personal image and the 
ornaments may interfere with each other. As a result, the composite image becomes unclear and does not look more 
interesting. 

35 [0010] An object of the present invention is to provide an image processing method, by which a personal image can 
be displayed more clearly, making the image look more interesting, and an art related thereto. 
[001 1] A first aspect of the present invention provides an image processing method comprising: detecting a position 
of a body part area in an input image; locating a coordinate system of an ornament according to the detected position 
of the body part area in the input image; and outputting an ornament-arranged input image, based on the located 

to coordinate system of the ornament. 

[0012] According to the method described above, when a person photographic object moves in an input image, an 
ornament can be moved according to the movement of the person photographic object. 

[001 3] A second aspect of the present invention provides an image processing method as defined in the first aspect, 
wherein the outputting comprises outputting an image data of the ornament-drawn input image. 
45 [0014] According to the method described above, it is not necessary to compose the input image and the ornament 
in a destination to which the image is outputted. 

[0015] A third aspect of the present invention provides an image processing method as defined in the first aspect, 
wherein the outputting comprises outputting an image data relating the ornament to the input image. 
[0016] According to the method described above, it is easy to restore the input image in a destination to which the 
so image is outputted, and it is convenient. 

[001 7] A fourth aspect of the present invention provides an image processing method as defined in the third aspect, 
wherein the ornament and the input image are related using a meta file. 

[0018] According to the method described above, it is possible to relate the input Image to the ornament concisely. 
[0019] A fifth aspect of the present invention provides an image processing method as defined in the fourth aspect, 
55 wherein a form of the meta file is SMIL. 

[0020] According to the method described above, relating the input image to the ornament can be easily performed. 
[0021] A sixth aspect of the present invention provides an image processing method comprising: detecting a size of 
a body part area in an input image; scaling a coordinate system of an ornament according to the detected size of the 
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body part area; and outputting an ornament-arranged input image, based on the scaled coordinate system of the 
ornament. 

[0022] According to the method described above, when a person photographic object gets closer to an image pick- 
up device, both size and movement path of the ornament become larger. On the other hand, when the person photo- 
5 graphic object moves away from the image pick-up device, both size and movement path of the ornament become 
smaller. Thereby, it is possible to make the ornament look like following the person photographic object that moves 
back and forth. 

[0023] A seventh aspect of the present invention provides an image processing method comprising: detecting a 
rotation amount of a body part area, the rotation being in-plane of an input image; defining a rotation amount of a 

10 coordinate system of an ornament according to the detected rotation amount of the body part area; and outputting an 
ornament-arranged input image, based on the defined rotation amount of the coordinate system of the ornament. 
[0024] According to the method described above, when a face is leaned aside, the ornament itself and the movement 
path of the ornament can be leaned correspondingly. Therefore, it is possible to make the ornament look like following 
the person photographic object that rotates in-plane of the input image. 

is [0025] An eighth aspect of the present invention provides an image processing method comprising: detecting a 
rotation amount of a body part area, the rotation being off-plane of an input image; defining a rotation amount of a 
coordinate system of an ornament according to the detected rotation amount of the body part area; and outputting an 
ornament-arranged input image, based on the defined rotation amount of the coordinate system of the ornament. 
[0026] According to the method described above, when a person does a "right-about", the ornament and the move- 

20 ment path of the ornament can be transformed according to the rotation of the person. Therefore, it is possible to make 
the ornament look like following the person photographic object that rotates off-plane of the input image. 
[0027] A ninth aspect of the present invention provides an image processing method comprising: detecting a body 
part area in an input image; defining opacity of an ornament, the opacity of the ornament in a foreground area with 
respect to the detected body part area being different from the opacity of the ornament in a background area of the 

25 ornament with respect to the detected body part area; and outputting an ornament-arranged input image, based on 
the defined opacity of the ornament. 

[0028] According to the method described above, when the movement path of the ornament comes to the back of 
the body part area, the ornament can be hidden. Therefore, it is possible to make the ornament look like surrounding 
the body part area and moving in three directions about the body part area. 
30 [0029] A tenth aspect of the present invention provides an image processing method as defined in the first aspect, 
wherein, when an ornament collides with a body part area, a moving direction of the ornament is altered so as to move 
the ornament away from the body part area. 

[0030] According to the method described above, a motion like reflection can be given to the ornament, when the 
ornament collides with the body part area. 
35 [0031] An eleventh aspect of the present invention provides an image processing method as defined in the first 
aspect, wherein a body part area is a face part area of a person as a photographic object. 

[0032] In the method described above, an image in which a face area is chosen as the body part area can be more 
interesting, since the face vividly shows the person's intention and feeling. 

[0033] The above, and other objects, features and advantages of the present invention will become apparent from 
40 the following description read in conjunction with the accompanying drawings, in which like reference numerals des- 
ignate the same elements. 

Fig. 1 is a functional block diagram, illustrating how an image processing apparatus functions according to the first 
embodiment of the present invention; 
45 Fig. 2 is a block diagram, illustrating the image processing apparatus according to the first embodiment of the 

present invention; 

Fig. 3 is an illustration, showing an outside view of the image processing apparatus according to the first embod- 
iment of the present invention; 

Fig. 4 (a) is an illustration, showing a meta file according to the first embodiment of the present invention; 
50 Fig. 4 (b) is an illustration, showing an input image file according to the first embodiment of the present invention; 

Fig. 4 (c) is an illustration, showing an ornament file according to the first embodiment of the present invention; 
Fig. 5 is a flowchart, illustrating the image processing apparatus according to the first embodiment of the present 
invention; 

Figs. 6 (a) to (c) are illustrations, showing templates according to the first embodiment of the present invention; 
55 Figs. 7 (a) to (c) are illustrations, showing the templates according to the first embodiment of the present invention; 

Figs. 8 (a) to (c) are illustrations, showing the templates according to the first embodiment of the present invention; 
Fig. 9 is an explanatory diagram, illustrating pattern matching according to the first embodiment of the present 
invention; 
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Figs. 10 (a) and (b) are block diagrams, illustrating the ornament information according to the first embodiment of 
the present invention; 

Figs. 11 (a) to (c) are illustrations, showing three kinds of display of a user interface according to the first embod- 
iment of the present invention; 

5 Figs. 12 (a) to (g) are explanatory diagrams, showing a process of the image processing apparatus according to 

the first embodiment of the present invention; 

Fig. 1 3 is an explanatory diagram, showing a coordinate system according to the first embodiment of the present 
invention; 

Figs. 1 4 (a) to (d) are explanatory diagrams, showing an animation according to the first embodiment of the present 
10 invention; 

Fig. 15 is an explanatory diagram, showing the animation according to the first embodiment of the present invention; 
Fig. 16 (a) is an illustration, showing an ornament according to the first embodiment of the present invention; 
Fig. 16 (b) is an explanatory diagram, showing an anchor point according to the first embodiment of the present 
invention; 

15 Fig. 1 6 (c) is an illustration , showing a composite image according to the first embodiment of the present invention; 

Fig. 17 is an illustration, showing the ornament according to the first embodiment of the present invention; 
Fig. 1 8 is an illustration, showing the composite image according to the first embodiment of the present invention; 
Fig. 19 is an illustration, showing an ornament according to the first embodiment of the present invention; and 
Fig. 20 is an illustration, showing the composite image according to the first embodiment of the present invention. 

20 

[0034] Hereinafter, a description is given of embodiments of the invention with reference to the accompanying draw- 
ings. 

[0035] Prior to detailed description of constructions, important terms are explained. 
[0036] A "personal image" is an image containing a part or a whole image of a person. 
25 [0037] Therefore, the image may be a whole-body image, a face image, an image of a sight of the person's back, 
or the upper half of the body. The image may also be taken with two or more people. Any kind of patterns, such as 
scenery and a design other than a person may comprise a background. 

[0038] A "body part" means a part of a person's body. The part of the person's body is included in the body part, 
when the part can be recognized as the part of the person's body, even when the body part is invisible because the 
30 skin of the body part is equipped with dress, a hat, or shoes. Therefore, a face is a body part and a head is also a body 
part. An eye, a nose, a mouth, an ear, eyebrows, hair, a neck, the upper half of the body with dress, a hand, an arm, 
a leg, feet with shoes, a head with a hat, eyes with glasses are also body parts. 

[0039] A "body part area" is an area defined as an area which the body part occupies in the personal image. The 
body part area may include, within itself, a part that is not the body part, and may be an area that is located inside the 
35 body part. 

[0040] Suppose that the body part is a face, the body part area may include an area in the circumference of the face, 
or the body part area may be a minimum-sized rectangle surrounding eyes, a mouth, and a nose, but not containing 
a forehead and ears. 

[0041] An "ornament" is an image pattern to be added. The image pattern may be an image pattern that is stored in 
40 advance, or may be an image pattern generated with computer graphics technology. The image pattern may be a 
pattern of a character, a pattern of a symbol, and a pattern of a figure. 

[0042] Fig. 1 is a block diagram, illustrating how the image processing apparatus functions according to the first 
embodiment of the present invention. As shown in Fig. 1 , the image processing apparatus of the present embodiment 
has the following components. 

45 [0043] An image input unit 1 acquires an input image. In the present embodiment, the input image contains a personal 
image which is permitted to move. The input image generally means an image for one frame of a moving picture. 
[0044] The input image may be an image data immediately after photographed with a camera, encoded data ac- 
cording to coding methods, such as MPEG, after photographing (the encoded data could be loaded from a recording 
medium or received from a communication device), or an image data obtained by decoding the encoded data. 

50 [0045] A control unit 2 controls each element shown in Fig. 1 according to the flow chart of Fig. 5. 

[0046] An image output unit 3 outputs an image that is composite of the input image and the ornament. The output 
image may be displayed on a display device, or may be encoded according to the above-mentioned coding method, 
and stored in a recording medium, or may be transmitted by other communication devices. 
[0047] An input image storing unit 4 stores an input image that the image input unit 1 inputted. 

55 [0048] A template storing unit 5 stores templates of a body part area. Hereinafter, a face area is regarded as a body 
part area in the present embodiment. 

[0049] As shown in Figs. 6 (a), (b), and (c), templates made by each modeled outline of face parts (a head, eyes, a 
nose, and a mouth) are prepared in different sizes for use. 
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[0050] As shown in Figs. 7 (b) and (c), templates with a certain in-plane rotation amount of an input image are used, 
besides the template of a basic position as shown in Fig. 7 (a). Thereby, the inclination of the face by rotation can be 
detected. 

[0051] As shown in Fig. 8 (b) and (c), templates with a certain off-plane rotation amount of an input image are used, 
5 besides the template of a basic position as shown in Fig. 8 (a). Thereby, the direction of the face can be detected when 
the direction of the face differs from the basic position (when the normal vector passing through the center of the face 
is not orthogonal to the plane of an input image). 

[0052] Using the templates of the template storing unit 5, a detecting unit 6 shown in Fig. 1 detects a body part area 

from the input image that is stored by the input image storing unit 4. 
io [0053] As shown in Fig. 9, the detecting unit 6 extracts an edge component, filtering the input image stored in the 

input image storing unit 4 with a differential filter. Moreover, the detecting unit 6 selects a template 53 from the template 

storing unit 5, and performs pattern matching using the selected template 53 and the edge component. 

[0054] The pattern matching is a processing which moves the template 53 (Nx * Ny pixels) over a search area (Mx 

- Nx +1) (My - Ny + 1) in an input image (Mx * My pixels) which is larger than the template, and searches the upper 
'5 left position (a, b) of the template at which a residual R given by the following equation becomes minimum. Here, the 

symbol indicates multiplication. 



20 [Equation 1] 



[0055] In Equation 1 , 1 (a, b) (mx, my) is a partial image of an input image, and T (mx, my) is an image of the template 



[0056] Instead of Equation 1 , the upper left position (a, b) of the template may be searched by calculating a cross- 
correlation coefficient C defined by Equation 2, and finding a maximum value for the cross-correlation coefficient C. 
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[0057] The detecting unit 6 uses various kinds of templates shown in Fig. 6, Fig. 7, and Fig. 8. When the detecting 
unit 6 uses Equation 1 , the detecting unit 6 finds a template with which the residual R becomes minimum. When the 

30 detecting unit 6 uses Equation 2, the detecting unit 6 finds a template with which the correlation coefficient C becomes 
maximum. The detecting unit 6 regards the values of position, size, inclination, and direction of the matched template 
as the values of position, size, inclination, and direction of the face image of the input image. 
[0058] In the case of a face image, the transition of the position, the size, the inclination, and the direction of the face 
may be obtained by performing the same processing for every frame or every several frames. 

35 [0059] When more than one face exist in the input image, the detecting unit 6 detects, as the face part area of a 
second person, a position of the lowest residual R or the highest correlation coefficient C within the input image area 
from which the already detected face part area as the first person is excluded. 

[0060] Similarly, for a face after a third person and on, the detecting unit 6 detects repeatedly, while setting up the 
low position of the residual R and the high position of the correlation coefficient C as the face part area in the area 
40 except for the detected face part areas, until the residual R becomes larger than a previously defined threshold or until 
the correlation coefficient C becomes smaller than a previously defined threshold. Thereby, face part area for two or 
more people is detectable. 

[0061] The above-mentioned detection need not always be performed for all the values. The detection of values 
except for the position and size of the face area may be optional. 
45 [0062] In Fig. 1 , an ornament information storing unit 7 stores ornament information. The ornament information is 
explained using Fig. 10. 

[0063] In an example (ornament information no. 1) shown in Fig. 10 (a), the ornament information consists of the 
ornament image data and a transform property, the ornament image data being related to the transform property. 
[0064] In the f ield of the "ornament image data", the ornament image data itself or an ID or a pointer which indicates 
50 the ornament image data itself is stored. As illustrated in Fig. 1 0 (a), the ornament image data may consist of only the 
image data, or have additional attributes (ex. behavior in a collision etc.). 

[0065] Stored in the field of the "transform property" is information how the ornament is moved with regard to the 
ornament layer and how the ornament layer is overlapped on the input image. 

[0066] Specifically, the ornament is defined on a local coordinate system (let an anchor point be the origin) by the 
55 transform property, the local coordinate system being different from the coordinate system of the input image. The local 
coordinate system has attributes of a position, a scale (rate of scaling), rotation, and opacity. 
[0067] In more detail, the anchor point shows a position and a direction of the origin of the local coordinate system 
concerning the ornament. The values set to fields of "position", "scale", and "rotation" indicate the position, the scale 
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(rate of scaling), and the rotational degree of the actual image on the coordinate system that the anchor point has (that 
is the local coordinate system). 

[0068] In an example (ornament information no. 2) shown in Fig. 10 (b), the ornament information consists of the 
ornament image data and a mask data, the ornament image data being related to the mask data. 
5 [0069] In this case, the "ornament image data" is data displayed as a background, and the "mask data" indicates a 
size and positional relationship of the data part of the background (shaded area) and the part in which the image of 
the face area appears (unshaded area). 

[0070] In Fig. 1 , an ornament arranging unit 8 defines the arrangement information of the ornament in accordance 
with the change of the body part area which the detecting unit 6 has detected. 
10 [0071 ] When a maximum matching is obtained with one of the templates shown in Fig. 7, the detecting unit 6 detects 
the rotation amount of the body part area, the rotation being in-plane of the input image; and the ornament arranging 
unit 8 defines the arrangement information of the ornament in accordance with the change of the rotation amount that 
the detecting unit 6 has detected. 

[0072] When a maximum matching is obtained with one of the templates shown in Fig. 8, the detecting unit 6 detects 
15 the rotation amount of the body part area, the rotation being off-plane of the input image; and the ornament arranging 
unit 8 defines the arrangement information of the ornament in accordance with the change of the rotation amount which 
the detecting unit 6 has detected. 

[0073] A detailed processing of the ornament arranging unit 8 is explained later, citing several kinds of examples. 
[0074] In Fig. 1 , a composing unit 10 outputs an ornament-arranged input image, in which an ornament is arranged 
20 in the input image based on the arrangement information that the ornament arrangement unit 8 has defined. The 
ornament-arranged input image is once stored in an output image storing unit 1 1 , and outputted from the image output 
unit 3. 

[0075] In order to perform displaying on a display device, the composing unit 1 0 generates image data in which an 
ornament is directly drawn to the input image (data form 1). As an image data for transmitting to a recording medium 

25 or to other communication devices, the composing unit 10 may preferably generate and transmit image data that an 
input image is related to parameters in drawing the ornament (data form 2). According to the data form 2, the input 
image can be easily restored, therefore it is advantageous. An explanation for the data form 2 will be supplied later. 
[0076] In Fig. 1 , an operation unit 12 is for a user to input operation information into the control unit 2. 
[0077] Next, construction examples for the image processing device shown in Fig. I and the pertaining peripheral 

30 parts are explained using Fig. 2. Fig. 2 is a block diagram of the above-mentioned image processing apparatus. 

[0078] The image processing apparatus shown in Fig. 2 exemplifies a camera-built-in cellular phone, in which the 
image processing apparatus shown in Fig. 1 is installed. As shown in Fig. 2, the camera-built-in cellular phone has the 
following elements. 

[0079] CPU 21 controls each element of Fig. 2 via a bus 20, and executes the control program that is recorded on 
35 ROM 23 and follows the flowchart as shown in Fig. 3. 

[0080] RAM 22 secures a temporary storage area that CPU 21 requires for processing. A flash memory 24 is a device 
equivalent to a recording medium. 

[0081] Moreover, a communication processing unit 26 performs transmission and reception of data with an external 
communication device via an antenna 25. 
40 [0082] An image processing unit 27 consists of an encoder/a decoder for a coding system, such as M PEG, processes 
the image that a camera 28 has photographed, and controls the display state of LCD 29 (an example of a display 
device) based on the image data that CPU 21 has directed. 

[0083] An audio processing unit 30 controls an audio input from a microphone 31 , and an audio output via a speaker 

32. 

45 [0084] An interface 33 is connected to the bus 20. A user can input operation information by a key set 34 via the 
interface 33, and can connect the other devices via a port 35. 

[0085] The function of an image input unit 1 is realized by a process in which CPU 21 or the image processing unit 
27 performs operation for data recorded in the flash memory 24 or data obtained by photographing with the camera 28. 
[0086] The functions of the control unit 2, the detecting unit 6, the ornament arranging unit 8 and the composing unit 
50 1 o are realized by processes in which CPU 21 exchanges data with RAM 22, the flash memory 24 and others, and in 
which CPU 21 performs operation for the data. 

[0087] Furthermore, the input image storing unit 4, the template storing unit 5, the ornament information storing unit * 
7, an arrangement information storing unit 9, and the output image storing unit 11 correspond to the storage areas 
secured by RAM 22, ROM 23 or the flash memory 24. The control unit 1 2 of Fig. 1 corresponds to the key set 34 of Fig. 2. 
55 [0088] CPU 21 performs recognition of the operation that a user performs on the key set 34, acquisition of the image 
from the camera 28, compression of a camera image and storage to the flash memory 24, loading and expansion of 
the storage image, image composition, image reproduction, and display on LCD 29 and so on. The image processing 
unit 27 may perform some items of the above-described processing. 
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[0089] Fig. 3 indicates the outside view of the image processing apparatus according to the present embodiment. 
As mentioned above, the image processing apparatus according to the present embodiment is installed in the camera- 
buirt-in cellular phone. 

[0090] The camera-built-in cellular phone of Fig. 3 is a flip-open type, which an upper casing 42 and a lower casing 
5 41 are connected foldably by a joint portion 43. Needless to say, the appearance of the image processing apparatus 

is arbitrary, and the present invention can be applied to devices such as PDA other than cellular phones. 

[0091 ] Explanation of the data form 2 is supplemented referring to Fig. 4. Assume that an ornament (file name "star. 

swT) shown in Fig. 4 (c) is to be related with an input image (file name "face.rm") shown in Fig. 4 (b). 

[0092] In this case, it is desirable to use the meta file in SMIL (Synchronized Multimedia Integration Language), as 
io shown in Fig. 4 (a). In Fig. 4 (a), a line <video src="face.rm"> and a line <animation src="star.swf">, which are held 

between <par> and </par>, are simultaneously executed. After all, the input image of Fig. 4 (b) and the ornament of 

Fig. 4 (c) are composed at the time of a display. 

[0093] In establishing relationship such as shown in Fig. 4, it is not necessary to create a composite image itself of 
the input image of Fig. 4 (b) and the ornament of Fig. 4 (c). 
15 [0094] Not only can the metafile in SMIL be used as described above, but also a file in XML (Extensible Markup 
Language) or an MP4 file and so on may be used. When the data form 2 is used, it is sufficient that the input image 
and the ornament are related by a certain form. 

[0095] Next, the flow of processing by the image processing apparatus according to the present embodiment is 
outlined using Fig. 1, Fig. 5, and Fig. 12. It is assumed that the ornament information (part 1) indicated in Fig. 10 (a) 
20 js used. 

[0096] In Step 1 , the control unit 2 confirms whether the processing has completed or not regarding alt the frames 
that constitute a moving image. When it has completed, the processing is ended. When it is not completed, the process- 
ing moves to Step 2. 

[0097] In Step 2, the image input unit 1 acquires the input image as shown in Fig. 12 (a), and the control unit 2 saves 
25 the input image in the input image storing unit 4. 

[0098] In Step 3, the control unit 2 orders the detecting unit 6 to perform detection for the face area. Thereby, the 

detecting unit 6 detects the face area using the template (Fig. 12 (b)) of the template storing unit 5. 

[0099] In Step 4, the control unit 2 orders the ornament arrangement unit 8 to determine arrangement information. 

Thereby, the ornament arrangement unit 8 uses the ornament information of the ornament information storing unit 7, 
30 defines arrangement information, and saves the arrangement information in the arrangement information storing unit 

9. Consequently, the ornament image on the ornament layer as shown in Fig. 12 (c) is acquired. 

[0100] In Step 5, the control unit 2 orders the composing unit 10 to perform composition of the input image and the 

ornament. Thereby, according to the ornament information saved in the ornament information storing unit 7, as shown 

in Fig. 12 (d), the composing unit 10 composes the ornament in the input image, and saves the resultant composite 
35 image in the output image storing unit 1 1 . 

[0101] In Step 6, the control unit 2 outputs the composite image of the output image storing unit 11 to the exterior 

via the image output unit 3. 

[0102] The control unit 2 repeats steps after Step 1 until the processing of all frames is completed. 
[0103] Next, an example of composition of a user interface is explained using Fig. 9. Here, the application chosen 
40 by the user is supposed to be image edit software, and is also supposed to be image composition mode that composes 
the ornament and the personal image photographed by the user. 

[0104] First, as shown in Fig. 1 1 (a), a personal image as the object to be edited is displayed on the LCD 29. Attaching 
to the personal image, an edit button 40 and a return button 41 are displayed. 

[0105] When a user chooses the edit button 40 with the key set 34, the list of the ornament that can be inserted is 

45 displayed, as shown in Fig. 11 (b). 

[0106] When the user chooses the item "tears" and pushes a determination button 43, the face area of the personal 
image will be detected, the ornament arrangement information suitable for the face area will be searched for, the 
ornament "tears" will be added to the input picture according to the ornament arrangement information, and the result 
will be displayed on the LCD 29, as shown in Fig. 11 (c). In Fig. 11 (c), the ornament of tears 50 and 51 are added to 

so the input picture. Here, the tears are inserted as moving animation as well as a still image. 
[0107] In the prior art, tears do not move and stay still where the tears are arranged. 

[0108] Even if animation is inserted, the movement of the teas in the animation is completely irrelevant to the move- 
ment of the persona] image. 

[0109] For example, in the prior art, tears keep running downward (even under the face), because tears are not 
55 automatically inserted right below the eyes, and tears are not automatically adjusted to stop running around the cheeks 
of the face. Moreover, when the face inclines or moves, the motion of the tears cannot be modified in accordance with 
the face. 

[0110] On the other hand, according to the present embodiment, tears are designed to be automatically inserted 
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right below the eyes, synchronizing with the movement of the body part area, because the location and size of the 
body part area (for example, the eyes within the face area in the present embodiment) are detected before tears are 
inserted. Moreover, the tears are able to follow the movement of the face. 

[01 1 1 J Hereinafter, the ornament arrangement information is further explained, while exhibiting application examples 
s according to the present embodiment. Fig. 13 is an explanatory diagram, illustrating an input image and a positional 
relationship of a coordinate system for the ornament. 

[01 12] A procedure that creates general animation is briefly explained as supplement before explaining the applica- 
tion examples. Basically, the animation, in which the ornament moves, is created by moving an ornament layer that is 
related to the transform property. 

10 [0113] Hence, in order to create animation, the attributes (position, rotation, etc.) of the transform are set to every 
key frame (existence located on a time-axis), and each value of the attributes Is changed according to the passage of 
time. Here, the key frame means a frame to which the values of the position and other properties of the layer at a time 
of criterion are attributed. When there are two key frames (their positions on time axe naturally differ) that have different 
attribute values, for any frame locating between the key frames in the time axes, each value of attributes can be 

15 determined by calculating a mean value regarding the frame. In short, animation can be created by repeating the 
operations of (1) creating the key frames and (2) setting up the move locus by interpolation between the key frames. 
[0114] Next, the procedure of creating animation by the above-mentioned technique is explained, using the animation 
shown in Fig. 1 4 as the example. Here, as shown in Figs. 14 (a) to (d), animation, in which an airplane flies from lower 
left side to upper right side, is created by changing the location, rotation, and scale of the layer. 

20 [01 1 5] In order to define the coordinate system that arranges an ornament layer, the anchor point of the coordinate 
system is set up first. In the example, as indicated in Fig. 13, the anchor point is set up at (X0, Y0) that is the center 
of the original image. It is also assumed that the directions of the coordinate axes are the same as the directions of 
the coordinate axes of the original image coordinate system. 

[0116] The anchor point is assumed to be fixed to the last of the animation. As a result, all values for the position 
25 and the rotation are decided on the coordinate system on which the anchor point is set. 

[0117] Next, the positions of the key frames and the move locus that connects the key frames are set up in the time 
axis. For example, as shown in Fig. 1 5, it is presupposed that the airplane is at point A in the first frame, then moves 
to point B after two seconds, and to point C after four seconds, and that the three points are connected with a curved 
line. Moreover, the rotation and the rate of enlargement and reduction are also set up to the three key frames that are 
30 the frames at the three points. A movement locus for a frame located between the key frames in the time axis is set 
up by the interpolation of the adjacent key frames. As a result, animation in which the airplane flies to the upper right 
comer can be created as shown in Fig. 14. 

(Application Example 1) 

35 

[0118] In the present example, after the position and range of the face part area are set up by template matching, 
the origin of the coordinate system (the anchor point) for the ornament is set at the point corresponding to the position 
of the face part area. 

[0119] Fig. 16 (a) shows an ornament data. Fig. 16 (b) shows how the origin of the coordinate system (the anchor 
40 point) for the ornament is set at the head part of the face part area. The ornament depicts a partially el liptical orbit. Fig. 
16 (c) shows the resultant composite image. 

[0120] When the persons face part area is detected, as shown in Fig. 16 (b), the anchor point is correspondingly 
moved to the head part of the face part area. Moreover, when the size of the face area changes, the scale of the 
coordinate system for the ornament is changed in proportion to the size of the face area. 
4£ [0121 ] When the direction of the face part area changes, the direction of the axis of the coordinate system is changed 
to the direction that is the same direction as in the first set-up. 

[0122] Thereby, even when a person moves relatively to a photographing equipment, the ornament follows the move- 
ment of the person, always drawing elliptical orbit on the head part, 

[0123] Moreover, in a plurality of the face images photographed in different scenes, the ornament can be moved on 
so the orbit, which is always proportional to the size of the head part, corresponding to each face position, range, inclination, 
and direction. As a result, it becomes more interesting. 

(Application Example 2) 

55 [0124] In the present example, the ornament information (part 2) is used, and the opacity of the ornament in the 
composite image frame is changed following the movement of the face area. 

[0125] The present example uses the ornament information (part 2) shown in Fig. 10 (b). The processing of Steps 
1 to 3 in Fig. 5 is performed in the same manner as the case where the ornament information (part 1) shown in Fig. 
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10 (a) is used. 

[0126] However, the processing in Steps 4 to 6 is different In Steps 4to 6, using the mask of Fig. 12 (e) f a still image 
shown in Fig. 12 (f) is visibly arranged in the shaded area, and the image detected as the face area within the input 
image is pasted in the unshaded area, after adjusting the size of the image of the face area to the size of the unshaded 
5 area. After the processing, the resultant composite image is outputted. 

[0127] Thus, the composite mask image with the still image as the background is used as the ornament arrangement 
information. According to the result of the face detection, the center of the composite mask image is arranged at the 
center of the face. The set-up of the opacity is controlled by the composite mask image when the still image and the 
input image are composed. 

io [0128] As a result, when composing with another image, the still image is prevented from overlapping with the face 
area. 

(Application Example 3) 

15 [0129] In the present example, as shown in Fig. 1 7, the depth information (Z-axis) regarding a viewpoint is added to 
both of the input image and the ornament. 

[0130] When the ornament is cut off in a virtual plane 60, which is equivalent^ the projection surface of the input 
image, the ornament is divided into a foreground area 61 that is located in the foreground side of the virtual plane 60, 
and a background area 62 that is located at the background side. 
20 [0131] Furthermore, the opacity of the foreground area 61 and the opacity of the background area 62 are changed 
and the two areas are composed. Thereby, as shown in Rg. 18, the animation, which the star goes around the face, 
can be created. 

[0132] What is necessary is to set intentionally the anchor point so that the locus passes the face. By making the 
background 62 that passes through the back of the face transparent, the ornament having the three-dimension-like 
25 motion can be added. 

(Application Example 4) 

[0133] In the present example, as shown in Fig. 19, the attribute (position and/or direction) of the ornament in the 
30 following frame is assumed by the attribute of the ornament in the present frame and a series of the attribute in several 
previous frames. For example, if the ornament is assumed to move at a constant and uniform speed, the attribute of 
the ornament can easily be assumed by a simple equation. 

[01 34] Furthermore, when the ornament overlaps with the body part area, the orbit of the ornament may be preferably 
changed. When plural ornaments are made to appear, each ornament is preferably allowed to move independently 
35 with each other. 

[0135] In the present example, as shown in Fig. 20, when the ornament collides with the border line of the body part 
area or the input image, the moving direction of the ornament is changed so as to move the ornament away from the 
border line. Thereby, when the ornament collides with the body part area, a reflection-like motion can be given to the 
ornament. As a result, the composite image becomes more interesting. 
40 [0/136] According to the present invention, even if both personal image and ornament are moving, the ornament can 
be designed not to interfere with the personal image, therefore, the personal image can be clearly displayed. Moreover, 
the composite image can be made more interesting by synchronizing the movement of the ornament with the movement 
of the personal image. 

[01 37] Having described preferred embodiments of the invention with reference to the accompanying drawings, it is 
45 to be understood that the invention is not limited to those precise embodiments, and that various changes and modi- 
fications may be effected therein by one skilled in the art without departing from the scope or spirit of the invention as 
defined in the appended claims. 



50 Claims 

1. An image processing method comprising: 

detecting a position of a body part area in an input image; 
55 locating a coordinate system of an ornament according to the detected position of the body part area in the 

input image; and 

outputting an ornament-arranged input image, based on the located coordinate system of the ornament. 
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2. An image processing method as defined in claim 1 , wherein said outputting comprises outputting an image data 
of the ornament-drawn input image. 

3. An image processing method as defined in claim 1 , wherein said outputting comprises outputting an image data 
5 relating the ornament to the input image. 

4. An image processing method as defined in claim 3, wherein the ornament and the input image are related using 
a meta file. 

10 5. An image processing method as defined in claim 4, wherein a form of the meta file is SMIL. 

6. An image processing method comprising: 

detecting a size of a body part area in an input image; 
15 scaling a coordinate system of an ornament according to the detected size of the body part area; and 

outputting an ornament-arranged input image, based on the scaled coordinate system of the ornament. 

7. An image processing method comprising: 

20 detecting a rotation amount of a body part area, the rotation being in-plane of an input image; 

defining a rotation amount of a coordinate system of an ornament according to the detected rotation amount 
of the body part area; and 

outputting an ornament-arranged input image, based on the defined rotation amount of the coordinate system 
of the ornament. 

25 

8. An image processing method comprising: 

detecting a rotation amount of a body part area, the rotation being off-plane of an input image; 
defining a rotation amount of a coordinate system of an ornament according to the detected rotation amount 
30 of the body part area; and 

outputting an ornament-arranged input image, based on the defined rotation amount of the coordinate system 
of the ornament. 

9. An image processing method comprising: 

35 

detecting a body part area in an input image; 

defining opacity of an ornament, the opacity of the ornament in a foreground area with respect to the detected 
body part area being different from the opacity of the ornament in a background area of the ornament with 
respect to the detected body part area; and 
40 outputting an ornament-arranged input image, based on the defined opacity of the ornament. 

10. An image processing method as defined in claim 1 , wherein, when an ornament collides with a body part area, a 
moving direction of the ornament is altered so as to move the ornament away from the body part area. 

45 1 1 . An image processing method as defined in claim 1 , wherein a body part area is a face part area of a person as a 
photographic object. 

12. An image processing apparatus comprising: 

so an input image storing unit (4) operable to store an input image; 

a template storing unit (5) operable to store at least one template of a body part area; 
a detecting unit (6) operable to detect the body part area out of the input image stored in said input image 
storing unit (4), said detecting unit (6) using the at least one template of the body part area stored in said 
template storing unit (5); 

55 an ornament information storing unit (9) operable to store ornament information; and 

an ornament arranging unit (8) operable to define ornament arrangement information, in harmony with change 
of the body part area detected by said detecting unit (6). 



11 



EP 1 434 170 A2 



1 3. An image processing apparatus as defined in claim 1 2, further comprising a composing unit (1 0) operable to output 
an ornament-arranged input image, based on the ornament arrangement information defined by said ornament 
arranging unit (8). 

5 14. An image processing apparatus as defined in claim 12, wherein said detecting unit (6) detects a rotation amount 
of a body part area, the rotation being in-plane of an input image; and 

wherein said ornament arranging unit (8) defines ornament arrangement information, in harmony with change of 
the rotation amount of the body part area detected by said detecting unit (6). 

10 15. An image processing apparatus as defined in claim 12, wherein, 

said detecting unit (6) detects a rotation amount of a body part area, the rotation being off-plane of an input image, 
and 

said ornament arranging unit (8) defines ornament arrangement information, in harmony with change of the rotation 
amount of the body part area detected by said detecting unit (6). 

15 

16. An image processing apparatus as defined in claim 12, wherein the ornament information comprises transform 
property and ornament image data, the transform property being related to the ornament image data. 

17. An image processing apparatus as defined in claim 12, wherein the ornament information comprises mask data 
20 and ornament image data, the mask data being related to the ornament image data. 

18. A cellular phone comprising: 

an input image storing unit (4) operable to store an input image; 
25 a template storing unit (5) operable to store at least one template of a body part area; 

a detecting unit (6) operable to detect the body part area out of the input image stored in said input image 
storing unit (4), said detected unit (6) using the at least one template of the body part area stored in said 
template storing unit (5); 

an ornament information storing unit (9) operable to store ornament information; and 
30 an ornament arranging unit (8) operable to define ornament arrangement information, in harmony with change 

of the body part area detected by said detecting unit (6). 
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Fig. 1 
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Fig. 3 
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Fig. 4(a) 

<SMIL> 
<head> 

<layout> . . . 
</ I ayout> 
</head> 

<body> 

<par> 

<video sr efface. rm"> 
<animation src="star. swO 
</par> 
</body> 
</smi l> 
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Fig. 5 
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Fig. 7(a) Fig. 7(b) Fig. 7(c) 
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Fig. 10(a) 
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Fig. 12(c) Fi* 12(d) 
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Fig. 14(a) 
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Fig. 15 
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Fig. 18 
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Fig. 19 
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